12 research outputs found
Tight Integrated End-to-End Training for Cascaded Speech Translation
A cascaded speech translation model relies on discrete and non-differentiable
transcription, which provides a supervision signal from the source side and
helps the transformation between source speech and target text. Such modeling
suffers from error propagation between ASR and MT models. Direct speech
translation is an alternative method to avoid error propagation; however, its
performance is often behind the cascade system. To use an intermediate
representation and preserve the end-to-end trainability, previous studies have
proposed using two-stage models by passing the hidden vectors of the recognizer
into the decoder of the MT model and ignoring the MT encoder. This work
explores the feasibility of collapsing the entire cascade components into a
single end-to-end trainable model by optimizing all parameters of ASR and MT
models jointly without ignoring any learned parameters. It is a tightly
integrated method that passes renormalized source word posterior distributions
as a soft decision instead of one-hot vectors and enables backpropagation.
Therefore, it provides both transcriptions and translations and achieves strong
consistency between them. Our experiments on four tasks with different data
scenarios show that the model outperforms cascade models up to 1.8% in BLEU and
2.0% in TER and is superior compared to direct models.Comment: 8 pages, accepted at SLT202